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METHOD AND SYSTEM FOR GENERATING 
SENSORY DATA ONTO THE HUMAN 
NEURAL CORTEX 


CROSS REFERENCE TO RELATED 
APPLICATION 


This application is a division of U.S. patent application 
Ser. No. 09/690,571 entitled “Method And System For 
Generating Sensory Data Onto The Human Neural Cortex” 
filed on Oct. 17, 2000 and now U.S. Pat. No. 6,536,440. 


FIELD OF THE INVENTION 


The present invention relates to non-invasive methods and 
systems for generating sensory experiences within the 
human neural cortex. 


BACKGROUND OF THE INVENTION 


Aconventional technique for generating neural activity in 
the human nervous system requires surgical implants. The 
implants may comprise wires that cause electronic impulses 
to interact with some portion of the human nervous system, 
such as the human neural cortex, and thereby cause neural 
activity in the human neural cortex. Researchers have suc- 
cessfully mapped audio sensory data to the cochlear 
channel, and visual data to the visual cortex. 


Conventional invasive techniques have several draw- 
backs. First, surgical implants may cause patient trauma and 
medical complications during and/or after surgery. Second, 
additional or on-going surgery may be required, particularly 
if new technology is developed. 


SUMMARY OF THE INVENTION 


The present invention solves the foregoing drawbacks by 
providing a non-invasive system and process for generating/ 
projecting sensory data (visual, audio, taste, smell or touch) 
within/onto the human neural cortex. 


One embodiment of the system comprises a primary 
transducer array and a secondary transducer array. The 
primary transducer array acts as a coherent or nearly- 
coherent signal source. The secondary transducer array acts 
as a controllable, acoustical diffraction pattern that shapes, 
focuses and modulates energy from the primary transducer 
onto the neural cortex in a desired pattern. The secondary 
transducer emits acoustical energy that may be shifted in 
phase and amplitude relative to the primary array emissions. 


The pattern of energy is constructed such that each portion 
of the pattern projected into the neural cortex may be 
individually pulsed at low frequency. The system produces 
low frequency pulsing by controlling the phase differences 
between the emitted energy of the primary and secondary 
transducer array elements. The pulsed ultrasonic signal 
alters the neural timing in the cortex. Changes in the neural 
firing timing induce various sensory experiences depending 
on the location of the firing timing change in the cortex. The 
mapping of sensory areas of the cortex is known and used in 
current surgically invasive techniques. Thus, the system 
induces recognizable sensory experiences by applying ultra- 
sonic energy pulsed at low frequency in one or more selected 
patterns on one or more selected locations of the cortex. 

One of the advantages of the present system is that no 
invasive surgery is needed to assist a person, such as a blind 
person, to view live and/or recorded images or hear sounds. 

This brief summary has been provided so that the nature 
of the invention may be understood quickly. A more com- 
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plete understanding of the invention can be obtained by 
reference to the following detailed description of the pre- 
ferred embodiments thereof in connection with the attached 
drawings. 


BRIEF DESCRIPTION OF THE DRAWINGS 


FIG. 1 illustrates one embodiment of a system in accor- 
dance with the present invention. 

FIG. 2 illustrates one embodiment of a transducer system 
within the system of FIG. 1. 

FIG. 3 illustrates one embodiment of a process in accor- 
dance with the present invention. 

Use of the same reference symbols in different figures 
indicates similar or identical items. 


DETAILED DESCRIPTION 


FIG. 1 illustrates one embodiment of a system 120 in 
accordance with the present invention. FIG. 1 shows a visual 
portion 100 of the human cortex located in a person’s brain 
100A, such as for example, a vision-impaired person’s 
brain. The system 120 of FIG. 1 is used with the visual 
cortex 100 merely as an example and is not intended to limit 
the scope of the invention. Instead of or in addition to the 
visual cortex 100, the system 120 may be used to stimulate 
neural activity in other areas of the nervous system. For 
example, the system 120 may be used as is or modified to 
generate audio, taste, smell or touch sensations within the 
brain 100A. 


In FIG. 1, the system 120 comprises a receiving module 
110, a processing module 101, a signal generator 102, a 
reference signal generator 103, a transducer system 106, a 
first signal line 104 and a second signal line 105. The 
receiving module 110, processing module 101, signal gen- 
erator 102, and reference signal generator 103, may be 
referred to as, alone or in combination, a sensory data 
processing system. Various configurations of the system 120 
may be configured in accordance with the present invention. 
The system 120 may comprise other modules and compo- 
nents in addition to or instead of the modules and compo- 
nents shown in FIG. 1. 

In general, the system 120 receives, analyzes and transfers 
the sensory data 112 to the human brain 100A. The receiving 
module 110 receives sensory input data 112. Such data 112 
may comprise live video data captured by a video camera 
(not shown) which a vision-impaired person may not be able 
to see. The sensory data 112 may be live or recorded. The 
data 112 may be generated by other sources, such as for 
example a VCR, a DVD player, a cable broadcast, a satellite 
broadcast, an Internet connection, etc. 

The processing module 101 receives input data 101A 
from the receiving module 110 and formats or converts the 
data 101A. For example, analog input data from the receiv- 
ing module 110 may be digitized and/or converted into a 
neural firing time difference pattern. In one embodiment, the 
system 120 uses a technique that is reversed from a tech- 
nique disclosed in “Reconstruction of Natural Scenes from 
Ensemble Responses in the Lateral Geniculate Nucleus” by 
Garrett B. Stanley et al. in the Sep. 15, 1999 issue of the 
Journal of Neuroscience, which is hereby incorporated by 
reference in its entirety. 

Processed data 101B is transferred to the signal generator 
102. Based upon the data 101B, the signal generator 102 
generates a first signal 104A on the first line 104. The 
reference signal generator 103 generates a reference signal 
105A on the second line 105. Both signals 104A and 105A 
are transferred to a transducer system 106. 
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FIG. 2 illustrates one embodiment of a transducer system 
106 within the system 120 of FIG. 1. The transducer system 
106 includes a primary (or first) transducer array 200, and a 
secondary (or second) transducer array 202. An aperture 201 
with a distance “d”? separates the primary and secondary 
arrays 200 and 202. The distance 201 may be fixed or 
adjusted depending on the wavelength of energy emitted by 
primary array 200. In one embodiment, the distance 201 is 
equal to the wavelength of sound emitted by the primary 
transducer 200. 


The primary transducer array 200 may comprise one or 
more columns and rows of individually-controllable piezo- 
electric elements. The secondary transducer array 202 may 
also comprise a two-dimensional array of individually- 
controllable piezoelectric elements. 


In one embodiment, the primary and/or secondary trans- 
ducer array 200, 202 each comprise a thin sheet of metal, 
glass, plastic or ceramic material covered with a two- 
dimensional array of individually-controllable piezoelectric 
elements. Each element in the arrays 200, 202 may emit a 
unique signal. The arrays 200, 202 may or may not be flat 
and may be shaped to conform to a portion of the human 
head over which the transducer system 106 lays to provide 
better focusing. The layout of individual elements within 
each array 200, 202 can also be altered to provide better 
focusing, according to the shape of the area of the human 
cortex where signal 104A is to be projected. 

In one embodiment, the arrays 200, 202 comprise piezo- 
electric elements that are held together by a flexible material, 
such as plastic or rubber. This embodiment allows the arrays 
200, 202 to further conform to a portion of the human head 
over which the transducer system 106 lays to provide better 
focusing. 

The primary and secondary transducer arrays 200, 202 are 
arranged such that the primary array 200 acts as a source of 
coherent energy, while the secondary array 202 acts as a 
programmable diffraction grating. For example, the primary 
transducer array 200 may comprise a phased array of 
emitters, whereby the combined output of some or all of the 
emitters appears to the secondary transducer array 202 as a 
coherent acoustical signal source. The primary array 200 
may emit acoustical energy, thereby providing an acoustical 
implementation of projective holography. In one 
embodiment, the phase of one or more array elements in the 
primary array 200 is controllable to allow shaping of the 
energy received by the secondary transducer array 202. The 
primary and secondary arrays 200 and 202 may emit ultra- 
sonic energy at the same wavelength. 

The secondary transducer array 202 may comprise an 
array of emitters, where each emitter can be individually 
controlled for amplitude and phase relative to the energy 
emitted by primary transducer 200. Changes in signal ampli- 
tude and phase are driven by signal 104A. The secondary 
array 202 may provide focusing and low frequency modu- 
lation of phase differences and/or signal amplitude between 
the energy emitted by the arrays 200, 202. The modulation 
of phase differences and/or signal amplitude induces low 
frequency vibrations in the neurons of the visual cortex 100. 
The focusing effect is accomplished by the primary array 
200 acting as a coherent signal source, and the secondary 
array 202 acting as a controllable diffraction pattern, based 
upon signals 104A and 105A. 

Ultrasonic frequencies may accurately place signal pat- 
terns within the cortex. Interaction of emissions from the 
primary and secondary arrays 200, 202 projects an interfer- 
ence pattern (e.g., low frequency signals or pulses) in the 
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brain 100A. The projected interference pattern creates a 
highly defined pattern within the visual cortex 100 or 
another part of the human neural cortex. Each point in the 
pattern may have an individually pulsed low frequency 
amplitude that is used to modify neural firing times. 

Low frequency amplitude modulation combined with 
wavelength phase interactions from the primary and sec- 
ondary transducer arrays 200, 202 form a stimulus to acti- 
vate neurons in the visual cortex area 100 or another other 
part of the human neural cortex. By controlling the pattern 
of signal amplitude and phase shifts in secondary array 202, 
a wide range of patterns can be focused towards visual 
cortex 100 or any other region of the human cortex. Ultra- 
sonic signals altering neural firings are discussed in 
“Temporally-specific modification of myelinated axon excit- 
ability in vitro following a single ultrasound pulse” by 
Mihran et al. published by the Ultrasound Med Biol 1990, 
16(3), pp. 297-309 and “Transient Modification of Nerve 
Excitability In Vitro by Single Ultrasound Pulses” by 
Mihran et al. found in the Department of Electrical and 
Computer Engineering, University of Colorado, 1990, paper 
#90-038, which are hereby incorporated by reference in their 
entirety. 

Changes in the neural firing timing induce various sensory 
experiences depending on the location of the firing timing 
change in the cortex. The mapping of sensory areas of the 
cortex is known and used in current surgically invasive 
techniques. 

FIG. 3 illustrates one embodiment of a process in accor- 
dance with the present invention. In a process block 301, the 
receiving module 110 (FIG. 1) receives sensory input data 
112 from, for example, a video camera, VCR, DVD player, 
cable broadcast, satellite broadcast, and/or Internet connec- 
tion. The receiving module 110 outputs the data 101A to the 
processing module 101 (FIG. 1). 

In a block 302, the processing module 101 processes the 
input data 101A. As stated above, in one embodiment, the 
processing module 101 digitizes analog data 101A from the 
receiving module 110 and/or converts the data 101A into a 
set of neural firing time differences or a pattern. 

In a block 303, the signal generator 102 converts the firing 
time differences to a first signal 104A. For example, the first 
signal 104A may comprise an acoustical pattern, which 
comprises a plurality of amplitude and phase differences. In 
one embodiment, this conversion is accomplished by using 
known techniques in generating projective holograms. 
Acoustic holography is discussed in “Nearfield acoustic 
holography: I. Theory of generalized holography and the 
development of NAH” by J. D. Maynard et al. in the October 
1985 issue of the Journal of the Acoustical Society of 
America, which is hereby incorporated by reference in its 
entirety. 

In a block 304, the reference generator module 103 
generates a reference signal 105A, which provides a coher- 
ent signal source, onto the second line 105. In one 
embodiment, the acts described in blocks 303 and 304 occur 
substantially simultaneously. 

In a block 305, signals 104A and 105A are transferred to 
transducer system 106. The first signal 104A is transferred to 
the secondary array 202. The reference signal 105A is 
transferred to the primary array 200. 

In a block 306, the transducer arrays 200 and 202 project 
a focused interference pattern onto the human cortex. The 
shape of the interference pattern and the amplitude pulse rate 
for each portion of the pattern may be controlled through the 
signals transferred in block 305. Low frequency pulses are 
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derived from the interaction of the emissions from the 
primary and secondary arrays 200, 202. 


In a block 307, low frequency pulsing of different points 
of the projected ultrasonic energy modifies the firing timing 
of the neurons in the human nervous system (in this 
example, the visual cortex 100), thereby giving rise to 
perceived sensory experiences, such as visual images. Sen- 
sory data is mapped in the neural cortex as differences in 
neural firing times. Thus, altering the firing times in cortical 
neurons can generate sensory experiences. 

One advantage of the present system is that no surgery is 
needed to change neural activity causing a sensory experi- 
ence. 

Although the present invention has been described with 
reference to specific embodiments, these embodiments are 
illustrative only and not limiting. Many other applications of 
this present invention will be apparent in light of this 
disclosure and the following claims. 

What is claimed is: 

1. A method for projecting sensory data in a part of a 
human brain, the method comprising: 

emitting a first acoustical energy as a coherent signal 

source toward the human brain; 

producing a diffraction pattern for the first emitted acous- 

tic energy, the diffraction pattern being based on input 
sensory data; and 

altering neural firing timing in the brain with the diffrac- 

tion pattern. 
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2. The method of claim 1, further comprising: 


converting sensory data to a plurality of neural firing time 
differences; 


converting the neural firing time differences to an acous- 
tical pattern signal; and 


generating the diffraction pattern from the acoustical 
pattern signal. 
3. The method of claim 1, further comprising: 


generating a reference signal; and 


emitting acoustic energy as a coherent signal source based 

on the reference signal. 

4. The method of claim 1, further comprising converting 
analog data from a sensory data source to digital data. 

5. The method of claim 1, wherein the act of altering 
neural firing timing in the brain with the diffraction pattern 
comprises generating neural activity in the brain such that 
the brain perceives sensory experiences. 

6. The method of claim 1, wherein the diffraction pattern 
comprises a plurality of amplitude and phase shifts in 
relation to the first emitted acoustic energy. 

7. The method of claim 1, wherein the diffraction pattern 
comprises an interaction of the first emitted acoustic energy 
and a second emitted acoustic energy. 

8. The method of claim 1, wherein the diffraction pattern 
comprises a set of controllable low frequency pulses. 
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